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STUDENT ASSESSMENT: 

WHAT DO SUPERINTENDENTS NEED TO KNOW? 

The days when a public school superintendent could be success- 
ful by being a good organizational manager are over. Throughout the 
1990s and into the first decade of the twenty-first century a transformation 
of the superintendency has occurred where school leaders are called upon 
to be instructional leaders as well as managers. Increasingly, student 
assessment and accountability in curriculum and instruction are at the 
heart of district level educational leadership. More than ever before, being 
a superintendent means being knowledgeable in classroom assessment, 
school and district level assessment, and the management and analysis of 
student assessment data to make teaching and school decisions (Elmore, 
2000; Johnson, 2002). 

By the late 1990s nearly all states had implemented a state student 
assessment system with higher stakes for schools and school districts than 
ever before. And most recently, federal legislation in the No Child Left 
Behind Act (NCLB) of 2001 has increased the accountability pressure on 
school districts even more. This external call for student assessment and 
accountability has coincided with calls from within the educational lead- 
ership community for school leaders to be instructional leaders as well as 
organizational managers (Arter, Stiggins, Duke, & Sagor, 1993; Bern- 
hardt, 2004; Berube, Gaston, & Stepans, 2002; Cawelti, 2004; Leithwood, 
Aitken, & Jantzi, 2001; Popham, 2001; Reeves, 2004; Schlechty, 2001). 

Today’s superintendents themselves believe that assessment and 
accountability are among their most pressing instructional leadership con- 
cerns. In The 2000 American Association of School Administrators 
Ten-Year Study of the American School Superintendent (Glass, Bjork, & 
Brunner, 2000), a nationally representative sample of 2,262 superintend- 
ents was surveyed. When asked to rank the most important challenges fac- 
ing them in their position, their second- and third-highest rankings out of 
29 issues and challenges facing the superintendency were assessing or 
testing for learner outcomes and accountability, respectively. 

In the policy environment of the beginning of the twenty-first 
century, superintendents are caught between the need to reconcile assess- 
ment for accountability with assessment to support instruction. Though 
this struggle was brought more to public attention by NCLB, school dis- 
tricts and their superintendents were already actively reforming their 
assessment systems prior to NCLB. In the Public Agenda report. Rolling 
Up Their Sleeves: Superintendents and Principals Talk About What’s Need- 
ed to Fix Public Schools (Farkas, Johnson, & Duffett, 2003), superintend- 
ents in the focus groups and survey study of a national random sample of 
1,006 public school superintendents surveyed in summer, 2003, reported 
that they were working hard to implement academic standards and account- 
ability in their schools well before NCLB was enacted in 2002. 

On the eve of the implementation of NCLB, Popham (2000) 
sounded an alarm to school leaders about the misuse of standardized tests 
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to measure educational quality. He decried the overuse of standardized 
achievement tests, whether published by testing companies or custom 
developed by states. And he charged superintendents with the moral 
responsibility of halting the rush to equate student achievement test scores 
with successful instruction. Among his recommendations were to (a) pro- 
vide an intense and comprehensive assessment-literacy program for 
teachers and administrators; (b) offer carefully structured briefing ses- 
sions to educational policymakers regarding appropriate and inappropri- 
ate ways to evaluate schooling; (c) implement a meaningful assessment 
literacy program for parents; (d) encourage the creation of autonomous 
parent action groups; (e) review, under carefully controlled conditions, the 
actual high-stakes tests being used; and (f) devise and implement more 
valid credible evaluative schemes. 

In the study reported here we reviewed the literature on accounta- 
bility and assessment in order to design a questionnaire to survey superin- 
tendents across Wyoming on their existing and needed knowledge about 
student assessment. Results on the presence or absence of gaps in knowl- 
edge that is deemed important by respondents and/or the literature can be 
used by colleges and universities in W^yoming, and perhaps elsewhere, to 
improve superintendent certification programs. 

Accountability and Assessment 

Accountability, until recently, has simply meant test scores and 
using test scores to hold schools and teachers accountable for student 
learning. In contrast, holistic accountability or student-centered accounta- 
bility provides district level administration with a systematic process to 
examine teaching practices, curriculum, and leadership decisions (Reeves, 
2004). What does the superintendent need to know about student-centered 
accountability? 

Student-centered or holistic accountability is not something done 
to teachers. Student-centered accountability requires the superintendent to 
look at qualitative and quantitative data that represent all the extraordinary 
steps teachers and students take to improve learning. Reeves (2004) point- 
ed out that student-centered accountability relies on assessment gathered 
from the four categories of teaching, leadership, curriculum, and parent 
involvement. The superintendent must take the lead in defining a plan for 
the identification, collection, and reporting of relevant data that address 
these four categories. 

Ross (2002) suggested that superintendents and school boards 
needed to rethink accountability. Accountability should deliberately con- 
nect all key players in the schooling equation: students, parents, teachers, 
building leadership, and superintendents. Superintendents should be ask- 
ing questions not only about schools as a whole, but about specific princi- 
pals, teachers, and classrooms. He reported that in order for districts to be 
accountable they must align their internal assessments with state and 
national assessments, recognize the cause and effects of what the ongoing 
assessment data demonstrate, and deliberately connect all key players in 
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the schooling equation. It is the responsibility of the superintendent to 
ensure that all stakeholders involved in the schooling equation know and 
understand student assessment data and their individual role in the process 
of data collection, interpretation, and presentation. 

Recent literature related to the knowledge of student assessment 
needed by superintendents reflects three facets: planning, implementation 
of data-based decision making, and technical assessment knowledge. 
These are consistent with various leadership standards related to student 
assessment knowledge advocated by educational leadership professional 
organizations. 

Planning 

Mintzberg (1994) wrote about the need to define planning and 
then to answer the question, “why plan?” He defined planning as “a for- 
malized procedure to produce an articulated result, in die form of an inte- 
grated system of decisions” (Mintzberg, 1994, p. 12). He found that 
organizations plan in order to coordinate their activities, ensure that the 
future is taken into account, control others in the organization, and formal- 
ize rational thinking. Leithwood, Aitken, and Jantzi (2001) defined strate- 
gic planning as a way to build commitment among organizational 
members to a shared vision for the future. But for some districts the strate- 
gic plan creates unmanageably large numbers of priorities making it 
impossible to monitor and track improved results (Leithwood, Aitken, & 
Jantzi, 2001; Reeves, 2004). 

Typically, at least until recently, plans have been devised largely 
for the benefit of external sources, such as central office administrators or 
to meet state and federal mandates (Johnson, 2002). It is now better under- 
stood that superintendents need to start with the board-determined district 
goals and build a student-centered accountability plan around those goals. 
Reeves (2004) strongly advocated for no more than six goals. He argued 
the district leader should keep in mind the cardinal rule of measurement 
which states that it is more effective to measure a few things frequently 
rather than many things once a year. Schmoker (2001) supported this 
when he wrote, “keep it simple” (p. 121). 

The superintendent’s knowledge of student assessment should 
assist in the district’s planning and direction-setting activities. The super- 
intendent is responsible for freeing up the time and supporting the effort 
of teachers, parents, community members, and the leadership team in 
coming to consensus on establishing priorities and allocating the needed 
resources. Planning should ensure that all faculty members know their 
goals, areas of weakness for each goal, and their time to meet to plan for 
the improvement of instruction to reach their goals (Schmoker, 2001). 

The accountability plan should provide for objective measure- 
ment of performance and hold all stakeholders accountable for results 
(American Association of School Administrators, 2003). By using multi- 
ple measures of achievement, students, teachers, administrators, parents, 
the school board, and the community all share responsibility for school 
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performance (Reeves, 2004). Data collected from teaching practices, cur- 
riculum implementation, parent involvement statistics, and student 
achievement data must be linked to the six or fewer district goals. The 
monitoring system should be selective and focus only on those elements 
for which there is evidence of impact on important outcomes such as stu- 
dent achievement, teaching, parent involvement, and leadership (Leith- 
wood, Aitken, & Jantzi, 2001; Reeves, 2004). 

Cawelti (2004) studied four districts that demonstrated gains in 
student achievement among all student subgroups while serving substan- 
tial numbers of low-SES and/or students of color. His research team iden- 
tified five factors that contributed to the successes of these four districts. 
These factors were motivation by the state’s accountability plan; pressure 
placed on district leaders from community members; stronger leadership 
and focus from superintendents; district identification and adoption of 
instructional strategies that better supported student learning; and the 
state’s unwillingness to continue accepting low performance from student 
subgroups. Cawelti (2004) found that the net effect of all these changes 
was to carve out new roles for the superintendency. 

Holistic accountability is more than test scores. Reeves (2004) 
suggested that administrators consider the antecedents of educational 
excellence such as supporting best practice teaching strategies to include 
assessment, feedback, and collaboration. These strategies require the 
superintendent, as instructional leader, to facilitate the planning and 
implementation of a district vision, goals, and a student-centered account- 
ability plan. Superintendents must take every available opportunity to 
help principals and teachers use student assessment data effectively to 
plan for improving student achievement. This is the key reason for data 
collection. Ezarik (2002) suggested key data-gathering steps for districts, 
such as developing a district plan for a data-driven culture and conducting 
an audit to determine a data-use policy. She also stressed the importance 
of involving teachers in self-assessment, reflection, and staff development 
to meet the needs of students, based on needs identified in student 
achievement data. 

In sum, planning for the improvement of student achievement 
must involve the board of education, the superintendent, principals, teach- 
ers, parents, community members, and students. Decisions made in the 
planning process must include the use of quantitative and qualitative data 
collected from students, teachers, parents, and leadership. The focus of the 
plan must be at the classroom and student levels to support instruction and 
learning. “No significant increase in student achievement will be forth- 
coming unless students receive higher quality and more focused instruc- 
tion in their classrooms” (Cawelti, 2004, p. 10). 

While the board of education is ultimately responsible for student 
achievement, the superintendent is charged with taking the planning stage 
to the next stage of implementation. It is the superintendent’s responsibil- 
ity to ensure the plan gets implemented and that the results are communi- 
cated to the public. 
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Implementation of Data-Based Decision Making 

Research supports the use of data in making decisions regarding 
improvement of student achievement (Cawelti, 2004; DuFour, 1995; 
Lambert, 2003; Leithwood, Aitken, & Jantzi, 2001; Marzano, 2003; 
Popham, 2001; Reeves, 2004; Schmoker, 1996; Stiggins, 1997; Waddle, 
2002). Superintendents need to model the use of student assessment data 
to make decisions by including board members, principals, and teachers in 
the data driven decision-making process. Data can help superintendents, 
principals, parents, community members, and, most of all, teachers, to 
make decisions about instructional strategies, resources, and classroom 
teacher support. Superintendents need to advocate strongly for the use of 
student assessment data and not just the collection of data (Reeves, 2004). 

District administrators must understand the importance of their 
role in collecting, interpreting, and communicating student assessment 
data to all stakeholders. Questions to ask are: “how will we use data; why 
data; how is the district currently using data; and who has access to what 
data” (Bernhardt, 2004; Johnson, 2002). The superintendent is ultimately 
responsible for coordinating the data collection, regulation, and reporting 
of progress to the staff, board, and community. Kotter (1996) stated that 
district leaders must create a sense of urgency by continually examining 
student assessment data, ensuring that the faculty and community under- 
stand what the data are saying, and then leading the decision-making 
process to support continual improvement of student achievement. 

One example of the powerful effects of the use of data in support 
of instructional improvement was reported by Vail (1999). In Oxford 
School District, Mississippi, the initiative of a school board and new 
superintendent were instrumental in moving a district from “mediocre” on 
the state’s rating system based on achievement test scores to a Level 3, 
“successful,” and then to one of only 12 school districts in the state rated 
at that time as Level 5, “exceptional.” This occurred in only a 3-year peri- 
od beginning in 1993. According to Vail (1999), a key reason for the quick 
change was that the superintendent developed an assessment system for 
the district where individual student progress could be monitored. The 
superintendent “can crunch numbers with the best of them. He has stu- 
dents’ test scores on his computer where he can readily pull up test data 
and generate various reports and comparisons” (p. 26). As a result of close 
study of district assessment data, a radical change in the reading program 
was implemented and the assessment system was able to track individual 
student progress beginning at second grade. 

The superintendent should be aware of central office strategies 
proven to support student achievement. Schmoker (2001) suggested 
implementation of the fundamentals of improvement. These fundamentals 
included involving all staff in data analysis and goal setting, providing the 
time and tools to support effective teamwork, supporting district-based 
research and development, lessening the fear of state and standardized test 
results, using standardized tests results for the students’ benefit, creating 
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end-of-course assessments, conducting periodic assessments and data 
analysis, and empowering instructional teams. 

Data about all parts of the district need to be analyzed on a regu- 
lar basis in order to understand the entire system. Members of the school 
community must be able to gather, analyze, and accurately understand 
which strategies are not working and what to do differently to get better 
results (Bernhardt, 2004). Stakeholders need data analysis to lead to con- 
tinuous improvement, which requires measuring and evaluating school 
processes on an ongoing basis. Bernhardt (2004) suggested the implemen- 
tation of Continuous Improvement Continuums at the district level adapt- 
ed from the Malcolm Baldridge Award Program. The continuum provides 
an authentic means for measuring district wide improvement and growth; 
it provides an ongoing self-assessment of the district (Bernhardt, 2004). 
This process involves all staff, students, parents, and community stake- 
holders in the self-assessment. Results from the self-assessment are used 
to acknowledge accomplishments, to set goals for improvement, and to 
keep community partners apprised of district progress (Bernhardt, 2004). 

Access to the data is of utmost importance. Staff needs ready 
access to current data in order to analyze, report, and use data to make 
informed decisions about improving instruction and student learning. 
Bernhardt (2004) strongly advocated for a data warehouse. She explained 
that a data warehouse is designed to allow for the manipulation, updating, 
and control of multiple databases that are connected to one another via 
individual student identification numbers. Data warehouse systems are 
expensive, therefore, Bernhardt (2004) recommended spending time up 
front to decide how to select, implement, and maintain the data ware- 
house. This will be a major budget and staffing issue. 

In conclusion, effective use of data can support effective instruc- 
tion and learning to improve student achievement. Student-centered 
accountability provides a systemic examination of teaching practice, cur- 
riculum, and leadership decisions (Reeves, 2004). Superintendents are 
charged with the responsibilities of building a data-friendly culture, ensur- 
ing that school community members understand their roles and responsi- 
bilities, providing ongoing professional development that fosters new 
skills, and establishing a system focused on continuous improvement 
(American Association of School Administrators, 2003). Last, but not 
least, an extremely important piece to the student-centered accountability 
plan is to establish and implement celebrations of successes. 

Technical Assessment Knowledge 

The ability to plan assessment systems, to implement data-based 
decision making, to improve the classroom assessment used by teachers, 
and to communicate student assessment data requires technical knowl- 
edge in the area of student assessment. Even before NCLB there were 
calls for increased assessment literacy among teachers, principals, super- 
intendents, school boards, and the public. For example, Arter, Stiggins, 
Duke, and Sagor (1993) articulated a set of 12 assessment competencies 
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for principals and, by extension, superintendents. According to Arter et al. 
(1993), these instructional leaders should: 

1. Know the attributes of sound student assessment and how to 
apply them to the assessments used in the school building; 

2. Know the attributes of a sound student assessment system 
[italics added] and how to apply them to the assessment sys- 
tems used in the building; 

3. Know issues related to ethical and inappropriate use of 
assessment information and how to protect students and staff 
from misuses; 

4. Know the importance and features of assessment policies and 
regulations that contribute to the development and use of 
sound assessments at all levels of use; 

5. Know the importance of and be able to work with staff mem- 
bers to set specific goals for integration of assessment into 
instruction, and to assist teachers in reaching those goals; 

6. Know the importance of and be able to evaluate teachers’ 
classroom assessment competencies and build such evalua- 
tions into the supervision process; 

7. Know the importance of and be able to plan and present, or 
secure the presentation of, staff development experiences that 
contribute to the development and use of sound assessment at 
all levels of decision making; 

8. Know the importance of and how to use assessment results 
for instructional improvement at the building level; 

9. Know how to accurately analyze and interpret building-level 
assessment information; 

10. Be able to act effectively upon assessment information; 

11. Know and create the conditions necessary for the appropriate 
use of achievement information; and 

12. Be able to communicate effectively with all interested mem- 
bers of the school community about assessment results and 
their relationship to instruction, (p. 5) 

Stiggins (1997) supported the need for leaders to identify effec- 
tive instructional strategies and find the most effective instructional strate- 
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gies that have the biggest impact on student achievement. Superintendents 
must be assessment literate, they must know how sound assessment 
relates to quality instruction, and they must strive to maintain a balanced 
use of assessments (Stiggins, 1997). 

Impara, Plake, and Merwin (1994) sought to learn what the 
knowledge base for assessment among school administrators should be. In 
their study, a sample of 1,685 administrator members of the American 
Association of School Principals (AASA), the National Association of 
Secondary School Principals (NASSP), and the National Association of 
Elementary School Principals (NAESP) responded to a mailed survey in 
1993. The five areas of assessment knowledge and skills receiving the 
highest importance ratings by respondents were ability to (a) evaluate 
school or system assessment or testing programs, (b) be aware of changes 
in testing and assessment practices, (c) communicate test/assessment 
results to the media and general public, (d) develop a plan for assessment 
implementation for your building or school system, and (e) read the cur- 
rent literature on assessment. 

They rated their own level of knowledge and skills highest for (a) 
know the purposes of different kinds of testing, e.g., achievement, IQ, 
diagnostic; (b) know terminology found in reports for standardized tests, 
such as grade equivalent scores, percentile scores, and percentile bands; 
and (c) understand the concepts associated with testing, e.g., reliability, 
validity. 

The highest four ratings of need for knowledge were (a) under- 
stand the appropriate linkage between curriculum content and different 
kinds of tests; (b) know terminology found in reports for standardized 
tests, such as grade equivalent scores, percentile scores, and percentile 
bands; (c) know the purposes of different kinds of testing, e.g., achieve- 
ment, IQ, diagnostic; and (d) understand the concepts associated with test- 
ing, e.g., reliability, validity. 

Since the knowledge and skills they reported having and needing 
were so similar across superintendent, elementary, and secondary princi- 
pals, the authors concluded that “even though superintendents and princi- 
pals perform the tasks at different levels of emphasis or frequency, the 
amount of skill and knowledge they have and need to perform these tasks 
are very similar” (Impara et al., 1994, p. 526). Though the Impara et al. 
(1994) study was conducted nearly a decade before the implementation of 
NCEB, the need for this technical assessment knowledge and skill has 
only increased in the current policy environment of the school superinten- 
dent. 

Leadership Standards 

The need for knowledge of student assessment is also prominent 
in the standards of professional organizations. The Educational Eeader- 
ship Constituents Council (ELCC) created the most recent set of standards 
for both building level and district level leadership positions (Education 
Eeadership Constituent Council, 2002). The EECC Standards (2002) are 
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influencing certification requirements including testing, course work, and 
experiences for superintendents. The six ELCC standards focus on vision, 
culture, management, community, stewardship, and context, and are 
framed using knowledge, skills, and dispositions necessary in each of the 
standards. Candidates are asked to provide evidences of proficiency in 
each of the six standard areas. The importance of knowledge in student 
assessment and accountability systems is also prominent in these stan- 
dards. Three of these standards specifically address a candidate’s ability to 
demonstrate use of assessment data. 

Other professional organizations such as AASA, the Association 
for Supervision and Curriculum Development (ASCD), and NAESP have 
published explicit criteria for educational leaders in student assessment. 
Among the assessment-related responsibilities of superintendents fre- 
quently listed are (a) establishing procedures for collecting data about stu- 
dent learning and regularly sharing it with school staffs; (b) selecting and 
developing assessment instruments that are aligned with high standards 
for student learning; (c) communicating frequently about student learning 
to parents, the community, and media; (d) making student learning a pri- 
mary reference point for decision making and resource allocation; (e) 
developing policies, procedures, and culture where student assessment 
data are regularly used in visioning, goal setting, and planning for district 
improvement and accountability; and (f) developing and maintaining data 
management and reporting systems that support both internal and external 
needs for student assessment information. 

Superintendent Preparation 

Some critics have claimed that programs to prepare school system 
leaders have been inadequate. One observation about the superintendency 
made by Cooper, Eusarelli, Jackson, and Poster (2002) addressed this con- 
cern: 

Given their importance, as the top leaders of America’s school 
systems, the lack of scholarly attention paid to superintendent 
preparation speaks volumes about the difficulty of conceptualiz- 
ing and studying the process of educating, credentialing, and 
understanding the whole process: recruiting, educating, exposing 
superintendents to the concepts and practices for being successful 
leaders, and placing them in their new positions, (p. 243) 

In particular, there has been recognition that preparation of educa- 
tors at all levels in assessment of student learning has been inadequate. In 
the early 1990s there were calls for the need to increase the assessment lit- 
eracy among all educators (Popham & Hambleton, 1990). Stiggins (1995) 
reported that little progress had been made by 1995 in improving the 
assessment literacy of teachers, principals, and building administrators. He 
found that few colleges were training administrators adequately in class- 
room assessment and “few local or state agencies have in place the person- 
nel officers or building administrators who are capable of evaluating job 
candidates on their assessment capabilities or performance” (p. 239). 
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Indeed, there is considerable evidence that educational leaders are still not 
as well prepared to provide the leadership in assessment and accountabil- 
ity as they need to be (Black, 1998; Impara & Flake, 1996; Leithwood, 
Aitken, & Jantzi, 2001; Reeves, 2004; Levine, 2005). 

The professional literature and leadership standards present a 
daunting challenge to superintendents of today. Educational leaders are 
required to be knowledgeable not only in traditional areas of organization- 
al management, board and community relations, resource management, 
and personnel, but increasingly in newer areas of classroom assessment 
and accountability systems. Practicing school leaders and the programs 
that prepare them have been hard-pressed to keep up with increasing need 
for assessment and accountability knowledge. In order to keep the super- 
intendent preparation program at the University of Wyoming current with 
the needs of superintendents and to provide instruction based on best prac- 
tices, we sought to augment what we know from the professional litera- 
ture with more direct data from practicing superintendents from this small 
frontier state about how they rate their level of knowledge and the impor- 
tance of that knowledge. We have combined ideas from the literature with 
our findings to improve superintendent preparation and professional 
development in Wyoming. 

The Research Study 


Sample 


All the superintendents of the 48 Wyoming school districts were 
asked to participate in a survey. Thirty of 48 (62.5%) responded. Nine 
(30%) had a master’s degree as their highest degree; twelve (40%) had an 
Educational Specialist; and seven (23.3%) had a doctorate. Seven (25%) 
had been a superintendent 0-2 years, seven (25%) for 3-6 years, seven 
(25%) for 7-16 years, and seven (25%) for 17 years or longer. One report- 
ed having been a superintendent for 35 years. The mean number of years 
they had been superintendents was 9.3 (SD - 8.9), ranging between 2 and 
35. The mean district enrollment for our respondents was 2,168 students, 
ranging in size from 112 to 13,000 with a median enrollment of 800. 

Questionnaire 

Questionnaire development started with a review of the literature. 
Several studies (Arter, et al., 1993; Impara, et al, 1994; Impara & Flake, 
1996) were reviewed to support the development of the survey format. 
The review of literature revealed three major aspects of what superintend- 
ents know and need to know about student assessment: planning, imple- 
mentation, and technical assessment knowledge. Questionnaire items 
were developed based on these aspects. 

The survey consisted of 64 items, divided into four parts. Fart A 
included 28 statements addressing the question, “how important is it for 
superintendents to know X about assessment of student learning?” Super- 
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intendents responded using a 5-point scale ranging from “not important to 
know,” given a value of 1, to “extremely important to know,” given a 
value of 5. Part B requested participants to rate the same 28 statements 
using a 5-point rating scale to identify their own level of knowledge. The 
following rating scale was used for that section: 

1. Non-use: I am not aware of this area of knowledge. 

2. Awareness: I can describe, discuss, and/or explain this area of 
knowledge. 

3. Application: I have experience using this knowledge. 

4. Integration: I implement and practice this area of knowledge. 

5. Transfer: I have developed a sense of expertise in this area 
and I can train and mentor others in this area of knowledge. 

The statements were organized in sections related to planning, implemen- 
tation of data-based decision making, and technical assessment knowl- 
edge, consistent with the three constructs identified from the review of the 
literature. 

In Part C, the following five open-ended questions were present- 
ed to gain insight into the opinions of the respondents: 

1. What have we not listed that would be important for a super- 
intendent to know about assessment of student learning? 

2. What have we not listed that you currently know about stu- 
dent assessment? 

3. What should the superintendent’s role be in the area of stu- 
dent assessment? 

4. What would you like to know more about in the area of 
assessment? 

5. How do you stay current on “best practices” in the area of 
assessment of student learning? 

A demographic section concluded the questionnaire. Superintendents 
were asked to identify their highest degree, years of experience as a super- 
intendent, and their district’s enrollment. 

Procedure 

The questionnaire was administered by one of the researchers at 
the annual meeting of the Wyoming School Boards Association in 
November 2003. Explicit administration instructions were written in a 
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cover letter attached to the questionnaire. The purpose of the study and 
directions were explained prior to administration. The superintendents 
were asked to complete the survey and return it to the researcher or to take 
it with them and mail it to the researchers. 

Results 

Importance of Assessment Knowledge 

In the first part of the questionnaire superintendents were asked: 
“How important is it for superintendents to know X about assessment of 
student learning?” They were asked to rate 28 statements from “not 
important to know” (1) to “extremely important to know” (5). In general 
superintendents rated nearly ail 28 items as “important” or “extremely 
important” to know. Among the top seven highest rated items (see Table 
1) were two related to planning, where superintendents thought it very 
important to be able to use data and research in long-range planning for 
evaluating student improvement, and to determine what data need to be 
collected. The other five items were related to technical assessment 
knowledge including knowledge of the purposes of different assessments, 
testing terminology, and how to make valid interpretations of test results, 
interpret test scores for others, and communicate assessment information 
to others. 

Table 1 

Most Highly Rated Assessment Knowledge 


Questionnaire item 


Mean 


Use data and research methods to develop long-range plans 4.60 

for evaluating improvement in student achievement (Planning) 

Communicate assessment information to others 4.57 

(e.g. other administrators. Board of Education, principals, 
teachers, community members, and media) (Technical) 

Interpret test scores for others 4.47 

(e.g. colleague administrators. Board of Education, principals, 
teachers, community members, and media) (Technical) 

Know the purposes of different kinds of assessment 4.40 

(e.g. classroom, large scale, diagnostic) (Technical) 


Understand terminology found in reports from standardized tests 4.40 
(e.g. NCE scores, percentile scores, cut scores, and confidence 
intervals) (Technical) 

(continued) 
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Table 1 (continued) 


Questionnaire item Mean 


Understand variables that affect the valid interpretation of test 4.33 
results (e.g. student characteristics, curriculum match, and 
student motivation) (Technical) 

Determine what data to collect based on what is important 4.33 

to know about student performance, teacher quality, parent and 
community satisfaction, and district goals (Planning) 


Two of the lowest rated items were related to planning: lead the 
selection and development of assessments aligned with standards, and 
maintain data management and reporting systems (see Table 2). One item 
related to implementation: “Appropriately use student assessment infor- 
mation in personnel evaluation.” Four related to technical assessment 
knowledge: design assessments, reliability studies for assessments, a stan- 
dard-setting study, and understand alternative assessment techniques. 

Table 2 

Least Highly Rated Assessment Knowledge 


Questionnaire item 


Mean 


Design assessments (Technical) 3.10 

Design a consistency (reliability) study for district assessments 3.53 
and the assessment system (Technical) 

Design a standard-setting study for district assessments 3.63 

(Technical) 

Lead the selection and development of assessment instruments 3.67 
that are aligned with standards for student learning (Planning) 

Understand when alternative assessment techniques are 3.73 

appropriate (Technical) 

Maintain the data management and reporting systems, which 3.76 
support both internal and external needs for student assessment 
information (Planning) 

Appropriately use student assessment information in personnel 3.80 
evaluation (Implementation) 
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Level of Assessment Knowledge 

Next, superintendents were asked, “what do you know about 
assessment?” They were asked to rate the same 28 statements, now with 
reference to their own level of knowledge. Overall the responses were 
much more varied than for Part A, and the average ratings were lower. 
Among the planning items rated highest in knowledge level were: “Use 
data and research for long-range planning to evaluate improvement in stu- 
dent achievement,” “design a process for aligning district assessments 
with district and state standards,” and “establish procedures for collecting 
and sharing data about student learning regularly with school staff” (see 
Table 3). Only one item relating to implementation was rated highest, 
“evaluate the district assessment plan.” Four items related to technical 
assessment knowledge were rated at a high knowledge level: “Know the 
purposes of different kinds of assessments,” “understand standardized 
testing terminology,” “interpret test scores for others,” and “communicate 
assessment information to others.” 

Table 3 

Items Rated Highest in Knowledge Level 


Questionnaire item 


Mean 


Communicate assessment information to others 3.87 

(e.g. other administrators. Board of Education, principals, 
teachers, community members, and media) (Technical) 

Interpret test scores for others 3.83 

(e.g. colleague administrators. Board of Education, principals, 
teachers, community members, and media) (Technical) 

Understand terminology found in reports from standardi z ed tests 3.83 
(e.g. NCE scores, percentile scores, cut scores, and confidence 


intervals) (Technical) 

Know the purposes of different kinds of assessment 3.63 

(e.g. classroom, large scale, diagnostic) (Technical) 

Establish procedures for collecting and sharing data about 3.57 

student learning regularly with school staff (Planning) 

Evaluate the district assessment plan (Implementation) 3.57 

Design a process for aligning district assessments with district 3.53 

and state standards (Planning) 

Use data and research methods to develop long-range plans 3.53 


for evaluating improvement in student achievement (Planning) 
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Among the planning-related items, three were rated at superin- 
tendents’ lowest levels of knowledge: “Lead the selection and develop- 
ment of assessment instruments aligned with standards,” “develop data 
management and reporting systems,” and “maintain data management and 
reporting systems” (see Table 4). One item related to implementation was 
rated lowest, “appropriately use student assessment information in per- 
sonnel evaluation.” Three items related to technical assessment informa- 
tion were rated lowest in knowledge level: “Design assessments,” “design 
standard-setting studies for district assessments,” and “design reliability 
studies for district assessments.” 

Table 4 

Items Rated Lowest in Knowledge Level 


Questionnaire item 


Mean 


Maintain the data management and reporting systems that 2.60 

support both internal and external needs for student assessment 
information (Planning) 

Design a consistency (reliability) study for district assessments 2.60 
and the assessment system (Technical) 

Appropriately use student assessment information in personnel 2.77 
evaluation (Implementation) 

Design a standard-setting study for district assessments 2.77 

(Technical) 

Design assessments (Technical) 2.77 

Develop data management and reporting systems that support 2.87 
both internal and external needs for student assessment 
information (Planning) 

Lead the selection and development of assessment instruments 2.90 
that are aligned with standards for student learning (Planning) 


Difference Between Importance and Level of Knowledge 

An analysis of the difference between ratings of the importance of 
knowledge and the level of knowledge was conducted. In all cases the 
importance rating was higher than the knowledge level rating. The three 
planning items that had the largest difference between rating of impor- 
tance and level of knowledge were “maintain the data management report- 
ing system,” “use data and research methods to develop long-range plans 
for evaluating improvement in student achievement,” and “develop a data 
management and reporting system,” (see Table 5). Four items related to 
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implementation were among those with the largest discrepancy. These 
were “provide training in the development and use of performance assess- 
ments,” “identify weaknesses in instruction using student assessment 
information,” “use computer technology in the assessment and accounta- 
bility system,” and “appropriately use student assessment information in 
personnel evaluation.” Only one item relating to technical assessment 
knowledge appeared here. This was “design a body of evidence system for 
high school graduation.” 

Table 5 

Largest Differences Between Ratings of Importance and Knowledge Level 


Questionnaire item Mean 

difference 

Maintain the data management and reporting systems that 1.20 

support both internal and external needs for student assessment 
information (Planning) 

Provide training in the development and use of performance 1.13 
assessments (Implementation) 

Identify weaknesses in instruction using student assessment 1.10 

information (Implementation) 

Use computer technology in the assessment and accountability 1.07 
system (Implementation) 

Design a body of evidence system for high school graduation 1.07 
(Technical) 

Use data and research methods to develop long-range plans for 1.07 
evaluating improvement in student achievement (Planning) 

Appropriately use student assessment information in personnel 1.03 
evaluation (Implementation) 

Develop data management and reporting systems that 1.00 

support both internal and external needs for student assessment 
information (Planning) 


Items with the lowest discrepancy between importance of knowl- 
edge and level of knowledge included three items related to planning (see 
Table 6). The lowest discrepancy between importance of knowledge and 
level of knowledge were: “designing a process for aligning district assess- 
ments with standards,” “identifying student-learning activities to align 
with assessments,” and “establishing procedures for collecting and shar- 
ing student learning data regularly with school staff.” Only one item relat- 
ed to implementation appeared here, “appropriately using assessment 
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information in student grades.” Four items relating to technical assess- 
ment knowledge were rated low in discrepancy. These were: “understand- 
ing standardized testing terminology,” “designing assessments,” 
“understanding when alternative assessment techniques are appropriate,” 
and “interpreting test scores for others.” 

Table 6 

Smallest Differences Between Ratings of Importance and Knowledge Level 


Questionnaire item 


Mean 

difference 


Design assessments (Technical) 

0.33 

Understand when alternative assessment techniques are 
appropriate (Technical) 

0.53 

Appropriately use assessment information in student grades 
(Implementation) 

0.57 

Understand terminology found in reports from standardized tests 
(e.g. NCE scores, percentile scores, cut scores, and confidence 
intervals) (Technical) 

0.57 

Establish procedures for collecting and sharing data about 
student learning regularly with school staffs (Planning) 

0.60 

Interpret test scores for others 

(e.g. colleague administrators. Board of Education, principals, 
teachers, community members, and media) (Technical) 

0.63 

Identify student-learning activities that align with district and 
state assessments (Planning) 

0.67 

Design a process for aligning district assessments with district 
and state standards (Planning) 

0.67 


Open-Ended Questions 

We asked several open-ended questions. The first of these was, 
“what have we not listed that would be important for a superintendent to 
know about assessment of student learning?” A recurrent theme found in 
superintendents’ comments was the idea that what a superintendent needs 
to know is somewhat dependent on district size. Several pointed out that 
superintendents of large districts have a larger staff and are able to dele- 
gate many of the assessment-related activities. But superintendents of 
small districts must know it all. Other important knowledge themes iden- 
tified by superintendents were: understanding the difference between high 
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stakes testing and testing for achievement growth; recognizing the 
changes necessary because of NCLB; being able to communicate with the 
public in a way that all stakeholders can understand; knowing legal 
requirements for schools related to assessment; and learning how to lead 
building administrators who are leading teachers in proper use of assess- 
ments. 

When asked “what have we not listed that you currently know 
about student assessment,” comments were that not all assessments are 
appropriate for determining student achievement. Other comments stated 
that assessment is not an exact science, and we should consider unintend- 
ed and unwanted consequences and recognize the importance of growth 
by individual students. 

When asked, “What should the superintendent’s role be in the 
area of student assessment?” the major theme of comments was that the 
role of the superintendent depends on district size. Another question was, 
“what would you like to know more about in the area of student assess- 
ment?” The 12 responses had no common theme, but for the most part 
repeated specific ideas included on the questionnaire. 

The last question asked superintendents, “How do you stay cur- 
rent on ‘best practices’ in the area of assessment of student learning?” The 
most predominant sources were reading professional journals and other 
documents (n = 21), followed by attending workshops or training sessions 
(n = 15). Several mentioned discussions with other professionals (n = 6), 
and working with other professionals (n = 6). 

Three strong themes emerged from the comments made by 
respondents. One theme was that the superintendent must be a leader with 
vision who can lead building administrators to lead instruction. The sec- 
ond theme that emerged was that the superintendent must be the director 
and chief executive responsible for making sure the system works. The 
third major theme was that the superintendent must be a facilitator and 
leader of change. 


Discussion 

It must be kept in mind that the survey was a self-report of degrees 
of importance and knowledge by superintendents. It may be that superin- 
tendents who say they do not know enough about assessment should not be 
expected to know what they should know. And questionnaires like ours are 
prone to a “social desirability” bias, where respondents fall prey to overrat- 
ing their knowledge or the importance of an assessment issue. On the other 
hand, discrepancies between ratings of importance and knowledge are less 
prone to this tendency. Furthermore a focus on the relative ranking of items 
helps to control for this potential bias. 

The sample of W^yoming superintendents rated the importance of 
all 28 assessment knowledge items as high or very high. And, consistent 
with Impara et al. (1994), Wyoming superintendents rated the importance 
of assessment knowledge higher than they rated their own knowledge 
level. Though Wyoming superintendents rated the importance of assess- 
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ment knowledge higher than they rated their own knowledge level, there 
was important variation in the knowledge level across different topics. 

The traditional elements of basic statistical analysis, concepts of 
reliability and validity, and interpretation of various types of test scores 
are no longer sufficient. Instead, superintendents now need to have 
advanced and complex knowledge of assessment. They must be able to 
lead the design and implementation of an assessment system that at once 
provides real-time data to the district and school leaders and also meets 
external accountability demands. They also need to know how to design 
and maintain assessment data management systems. 

An important discrepancy between need for knowledge and rating 
of knowledge level was in using student assessment information to identi- 
fy weaknesses in instruction. A major challenge to educators is that once 
data help them to recognize that a sub-group of students is underperform- 
ing, the important next step is to find out why. Superintendents need to 
learn how to take this next step. Methods of qualitative and action 
research must be part of a superintendents’ knowledge related to assess- 
ment, accountability, and data-driven decision making. 

When we think of assessment knowledge we tend to focus on the 
technical aspects of measurement and testing. But these superintendents 
gave us the strong message that while superintendents need to have this 
basic knowledge, they increasingly need more complex knowledge. They 
need more knowledge of what data to gather, and particularly how to man- 
age these data in such a way that they serve the accountability and deci- 
sion-making needs of the district. Computer hardware and software, and 
system development are an integral part of the assessment and accounta- 
bility picture, and superintendents need more knowledge about these. 

Recommendations 

A brief survey of six schools in and near Wyoming, with doctoral 
programs in educational leadership, revealed only one with explicit 
coursework related to use of student assessment data and data-driven deci- 
sion making. And in this one program, the course description does not 
reflect a heavy emphasis on helping candidates to develop the knowledge 
found important in our study. These programs must place more emphasis 
on the use of student assessment data in decision making. 

Undoubtedly, university programs have the responsibility to 
match the appropriate knowledge to participants in educational leadership 
programs. The knowledge and skills of a superintendent in the area of stu- 
dent assessment appear to be significandy important to superintendents. 
The understanding of student data and the capability for leaders to use 
data to make decisions for the school and district must be at the heart of 
what preparation programs focus on. 

Practicing superintendents should be provided with opportunities 
to learn more about student assessment, understanding data, and making 
decisions directed by the data. The Wyoming Department of Education 
has provided some training opportunities but more could be offered. 
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The relationship among leadership standards, certification 
requirements for the superintendency, and university superintendent 
preparation programs is at the heart of the conversation about the superin- 
tendency itself. Specific questions are: (a) What is the relationship among 
certification, standards, and university programs as related to superinten- 
dent preparation? (b) What should the certification requirements be, if 
any? (c) How should university superintendent programs change? (d) 
What do alternatives to university programs look like? 

The professional literature and standards provide strong guidance 
for what preparation those training to become superintendents should 
receive in their graduate programs. High among these should be a solid 
foundation in principles of effective classroom assessment. Those topics 
which superintendents in our study rate as most important to know should 
also be an essential part of the curriculum and practicum experience for 
superintendent training programs. It may be that those rated less important 
to know should not be emphasized in coursework. And finally, those with 
the highest discrepancies should likely be the focus of professional devel- 
opment for practicing superintendents. 
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